The RoboCup-98 Teamwork Evaluation Session: A Preliminary Report

Increasingly, agent teams are used in realistic and complex multiagent environments. In such environments, dynamic and complex changes in the environment require appropriate adaptation of the teamwork (collaboration) among team-members. As RoboCup proposes to provide multi-agent researchers with a standard test-bed for evaluation of methodologies, it is only natural to use it for investigating this essential capability. During the RoboCup-98 workshop and competition a unique event took place: a comparative evaluation of the teamwork adaptation capabilities of 13 of the top competing teams. An evaluation attempt of this scale is a novel undertaking, and presents many novel challenges to researchers in the multi-agent community. This preliminary report describes the data-collection session, the experimental protocol, and some of the preliminary results from analysis of the data. Rather than proposing solutions and well understood results, it seeks to highlight key challenges in evaluation of multi-agent research in general, and of teamwork in particular..